Java Cluster readme and help file

For further information contact: Martin.Irman@mail.mcgill.ca (if you would like to use the below-described software – feel free to contact me)

Java Cluster is a cluster environment written in Java for automating the execution of Matlab simulations on multiple computers (currently used on multiple Windows XP computers). It does not facilitate automatic parallel execution of a Matlab script (that would be nice, wouldn’t it :) but allows the user to schedule the execution of a single Matlab script for hundreds of runs automatically on multiple computers. The environment supports the execution of multiple parallel “jobs” (a “job” is a Matlab script that should be executed x times) where the executions will be scheduled according to job priorities (just, priority and idle scheduling possible). The environment further provide tools to combine the data generated in the multiple executions.

The automatic execution of other than Matlab simulation jobs would be possible with some modification to the cluster software.

This documentation file contains these parts:

1. Cluster Environment
2. Installing a client on your machine
3. Installing full cluster software on a machine
4. Using the Cluster
5. Example of a script
6. Troubleshooting
7. Release notes

Do not move or rename this document. Use MsWord to edit.

Cluster Environment

The cluster environement consists of 3 componenet:

• JobServer - the server component that is running on all cluster machines and is responsible for actually executing the simulations
• ArbiterServer - a server that HAS to run on exactly one machine of the cluster. This server is responsible for scheduling jobs on the JobServers.
• ClusterClient - this is the user application where the user is able to start and manage simulations. It connects to the ArbiterServer.

To be able to manage simulations from your computer you only have to run the ClusterClient. You need to set it up to connect to the computer running the ArbiterServer (this can be the same computer).

Installing a client on your machine

• copy the distributed files in a directory on your computer
• Open 'cluster.properties' and set the ARBITER value to the name and port of the computer running the arbiter server
• Add the \matlab subdirectory of the cluster data to the default matlab path (this way you will have not problems running jCombine)

Installing full cluster software on a machine

(JobServer, ArbiterServer, ClusterClient)

Things that have to be set-up:

(you might try to use the setup scripts in the install_scripts folder - but

these are matched to a specific computer configuration)

1. Review cluster.properties:

* The matlab path needs to be set

* Comment out the STANDBY property if you don't want your computers

* to go to standby automatically

2. If you have setup jour JobServer computers to go to stand-by automatically

don't forget to setup Windows so that the computer automatically wakes up

on network activity

3. Review install.bat. The batch file includes system specific information.

* Change the account information for the "schtasks" commands

4. Add the java_cluster\matlab folder to the Matlab path!!!

5. Run install.bat to set-up automatic start of JobServer and/or

ArbiterServer on start-up of the computer.

6. You can do this manually without running install.bat

7. If e-mail notification is desired, you have to set-up ‘blat’ with e-mail account

information. The ‘blat’ is a command line utility that can send e-mails. It is

included in the ‘system’ folder (detailed documentation is also included). To

set it up run the following command from the command line:

blat –install <mail_server> <sender_email> 3 25 cluster <login> <password>

This only needs to be done once. As McGill use the following to this utility up:

Blat –install mailhost.mcgill.ca Martin.Irman@mail.mcgill.ca 3 25 cluster <DAS USERNAME> <DAS PASSWORD>

Now, user e-mail addresses have to be added to cluster.properties. Create a line:

Martin = Martin.Irman@mail.mcgill.ca

For every user. If this line is not added to the cluster.properties file for a user

the user will not receive e-mail notification.

8. (not important) Review eject.bat and load.bat in the \system folder and correct

the CD-ROM drive letter if you want to be able to command the CD to eject

Using the Cluster

Logging in:

• To log in into the cluster client just specify your name – no password is checked. The name is used for justly allocating computing power to multiple users and to prevent accidental manipulation of other user’s data.

How to run a simulation:

• Write a Matlab script that implements the simulation you want to run on the cluster. This script can use other scripts and functions that reside in the same directory (or subdirectory … but when calling such a function, the path has to be provided to matlab) as the original script that will be given to the cluster. When sending the script to a cluster node for execution the whole folder where the script resides will be zipped and sent to the computing node (thus avoid storing large amounts of data in the same folder).
• The script should write results into the results subfolder of where it resides. The commands used can be like this:

save results\matlab.mat

save results\matlab.mat a_variable b_variable

save results\ber.mat snr ber

Careful! The following does not work (Matlab does not like the leading backslash):

save \results\matlab.mat (DOES NOT WORK!)

This commands will save the current workspace (or the specified variables) in the file matlab.mat (or ber.mat). Later everything that is generated inside the results folder is moved back to the client where it can be analyzed using the jCombine command

• Open the Cluster Client software and perform the following steps to run 100 executions of a rnd_num.m script:
• Click on the ‘Open’ button and select the ‘rnd_num.m’ script
• Click on the ‘Package’ button to zip the folder with the script. This package will appear in the package window of the application and it can be executed. A version number will be appended to the package name.
• A ‘Execute’ dialog will automatically come up: specify the number of simulations: 100
• Now the executions of the package are running on the computing nodes. You can control the execution by using the ‘Execute’, ‘Stop’, ‘Delete’ package commands (move over the buttons to get tool-tip help).
• To pause an execution: click the ‘Execute’ button and specify 0 as the priority.
• The generated data from the results folders of the simulations will be downloaded back to a so called arbiter server and from that arbiter server to the client system (the zipped files will be stores in the cluster installation folder in the subdirectory ‘cluster_client\results’).
• To analyze the generated data either use the ‘Combine’ button and specify what file in the results folders should be used and what variable (variables) obtained or use the jCombine command directly in Matlab. To get help on how to use the jCombine: open Matlab and type ‘help jCombine’.

What is priority:

• If priority is 0 the package will not be scheduled (execution is paused).
• Packages with positive priorities are time-sliced. Each package gets a fair share of the time of the cluster computers where packages with higher priority get more time. If all the priorities are equal, all packages will be allocated the same amount of time.
• If the priority is less then zero – these packages will not be time-sliced. The package with the lowest negative priority is executed first. When this one is finished the one with the next lowest priority is executed. These packages are executed prior to the time-sliced packages if the priority is less than -1000 and after all the time sliced packages have finished if the priority is greater than -1000.
• Thus if packages with the priorities -1010, -1009, -1003, 100, 200, -5, -3, -1 are active than this packages will be executed exactly in this order while the packages 100 and 200 will be executed concurrently (the 200 package will be granted 2x the computing time of the 100 package).

Controlling the cluster:

• To check on the various computing nodes click the ‘Servers’ button. A window will open up where you can see what is executed on the computing nodes, you can reboot them, kill executions, …
• If you don’t wish to download the data to your local system but you want to see how your simulation performs (i.e. you are connecting from home using a modem): uncheck the ‘Grab Results’ checkbox when logging in into the cluster. Than data will remain on the arbiter server. If you decide that you want to download the data check the ‘Grab Results’ toggle button. (!Warning: once downloaded the data will be deleted from the arbiter server).
• ‘Delete’ your packages when you are finished with the simulations to save disk space. (!Warning: deletes all simulation results)
• Use ‘Get Results’ to retrieve results from the Arbiter without deleting them from there.

Users and package scheduling:

• Packages of one user are scheduled on the principle: oldest package first. After the older package finished all it’s runs the newer package starts to be scheduled.
• If multiple users are using the cluster a just (randomized) schedule is used so that these users are assigned the same computing power. Thus if user A runs a simulation that takes 10 minutes and user B runs a simulation that runs 20 minutes, the simulations of user A will be scheduled twice as many times as the simulations of user B.
• If you need to run multiple simulations concurrently (why cannot you wait till the first finishes and the second will start automatically?) you have to log in with two different usernames – yet, you are grabbing more computing power than you should! And that is unethical!

Email notification:

• Make sure ‘blat’ was set-up (refer to the installation section above).
• To enable e-mail notification your username has to be listed in the ‘cluster.properties’ file along with your e-mail address. A entry like the following has to be in the file:

Martin = Martin.Irman@mail.mcgill.ca

• Add a jSetCombine command to your simulation script. This command determines which variables will be included in the results file sent to you by e-mail.

jSetCombine(strMatlabFile,strVariable [,strName])

Call this function from a simulation .m file to set which variabls should

be combined automatically. Multiple jSetCombine statements can be

included in the simulation script. The command has to be issued while the

current derectory is either the results folder or it's parent folder.

Sets which variables should be combine if these are not explicitelly

specified in jCombine. Specify a matlab .mat file (strMatlabFile) and

which variable to combine (strVariable). Optionally specify the name for

the combined variable (strName) in the results. If not specified the

strVariable is used ... it needs to be specified if strVariable uses

wildcards i.e. '*'.

jSetCombine('matlab.mat','SNR');

jSetCombine('matlab.mat','*','all');

• Don’t turn on the ‘Grab Results’ option. When ‘Grab Results’ is on the results are copied to the client and deleted from the arbiter. The preparation of the results for e-mail notification is done on the arbiter and thus if this option is on some results might have already been deleted and they will not be present in the e-mail.

Blacklisting computers:

• If the computers in the cluster are not identical (for example some have less memory) we might want our script to execute only on some of them. To do this, there is a blacklisting mechanism. If the script that is running on a cluster computer determines that the computer is not good enough for it can exit with a special command that will prevent further executions of this script on this computer. The command to use is a Matlab function jBlacklist. Two other functions jSystemName and jSystemMemory are available to check system properties. Thus the following script part would ensure that the computer that should run this script has at least 600MB of physical memory:

% Blacklist computer if it does not have enough memory

% (physical memory in MB).

requiredMemory = 200;

if (jSystemMemory < requiredMemory)

% Construct en error message

strCause = [' Not enough memory on the system! present: ' num2str(jSystemMemory) ' < required: ' num2str(requiredMemory)];

% Blacklist the computer - this function terminates matlab execution!!!

jBlacklist(strCause);

end;

• Or the following would blacklist cluster5 computer:

if (strcmp(jSystemName,’cluster5’))

jBlacklist;

end;

• You can use jBreak to terminate the execution of a script without reporting an error (thus the script will be rescheduled on the computer).

Example of a script

% This is a example of a script to be used in a cluster simulation

% Blacklist computer if it does not have enough memory

% (physical memory in MB).

requiredMemory = 600;

if (jSystemMemory < requiredMemory)

% Construct en error message

strCause = [' Not enough memory on the system! present: ' num2str(jSystemMemory) ' < required: ' num2str(requiredMemory)];

% Blacklist the computer - this function terminates matlab execution!!!

jBlacklist(strCause);

end;

% Here would be the code of the simulation we want to execute (let's just

% generate a random matrix in this example).

% Generate a random matrix

randn('state',sum(100*clock));

a = rand(2,2);

% Let's wait a few seconds ... as if we would be doing something

pause(30);

% Save the workspace in the results folder - only files inside the

% results folder will be retreived back to the cluster client

save results\matlab.mat

% You can use jCombine('rnd_num','matlab.mat','*') to retreive the

% saved workspace or just jCombine('rnd_num','matlab.mat','a') to

% retreive the variable 'a' from all the 'matlab.mat' files.

Troubleshooting

1. Matlab cannot find jCombine:

* Add the \matlab subdirectory of your cluster data to the Matlab path!

2. Some computers have problems:

* Reboot the problematic computers

* Check if there is free space on the disk where the cluster data is

located. If the computer is low on disk space. Run ClusterClient and

use the 'Purge' button in the 'Servers' window. This will clean up old

data. Check if it helped.

3. Some computers are running a job that should have long finished but it’s

not finishing:

* Matlab does not exit if another instance of Matlab is running at the

same machine and this instance is unresponsive (running simulations).

As soon as this Matlab finishes or if you kill it manually also the

Matlab launched by the cluster will exit and the server will resume

normal operation.

4. An error is reported by the cluster client: ‘error saving file’:

* Make sure that you save jour data in the results forlder using the

command ‘save results\matlab.mat’ and not ‘save \results\matlab.mat’.

Matlab does not like the extra backslash.

5. Some computers are not good for running the simulations (they don’t have

enough RAM and the swapping could damage the disk):

* Blacklist the computers where you don’t want to run the simulation. If

a script determines that a computer is not good enough for the simulation

it can blacklist the computer from further executions of this script

by calling jBlacklist (refer to the description above).

6. Blacklisting does not work, the computer is listed as blacklisted but the

simulations start up on it anyway:

* Make sure that the name of the computer in the cluster.properties file

match the name that appears in the blacklist exactly. If not – change

the name in the cluster.properties file to match. For example: localhost:8188

does not match 127.0.0.1:8188, alpha4:80 does not match ALPHA4:80 which does

not match ALPHA4:0080!

7. ClusterClient or ArbiterServer is slow:

* Delete unused packages

* ‘Purge’ the cluster using the purge button in the ClusterClient servers

view. Note: this will erase all information about packages older than 30 days!

* Optimize the file system. The amount of files in the ‘results’ folders might

slow down the filesystem. You can speed-up the operation by disabling the

short filename generation by using the following command:

fsutil behavior set disable8dot3 1

Note: some older application that use the short (8,3) filenames might stop to work!

9. When I command the servers to eject the CD, the servers don't do that:

* Is the correct drive letter specified in the batch files eject.bat

and load.bat on the servers in the \system folder?

10.Computer goes to stand-by and never wakes up again:

* The wake-on-LAN capability has to be enabled to wake-up the computer after it was

sent to stand-by by the arbiter. Try pinging the computer / connecting via remote

desktop or otherwise disturb the computer – if it does not come on then there is

some problem with wake-on-LAN.

* Wake-on-LAN didn’t work with WinXP Service Pack 2 installed, when we uninstalled it

and installed SP1 only the feature worked.

* Bios or windows settings might be disabling this feature

* If nothing helps disable the goto-standby feature on the computer by commenting out

the STANDBY property in cluster.properties.

11.Arbiter cannot connect to JobServers / ClusterClient cannot connect to Arbiter:

* Check windows firewall settings. Optionally enable ports 8188, 8189, 8191.

12.ClusterClient shuts down when starting Matlab. A error message is generated by the

Java Virtual Machine that a exception occurred in the AWT package:

* We had a issue with an old ATI graphics card driver that caused this. Try installing

the newest drivers (do windows update – in our case it helped).

Release Notes

To do list:

• Set all cluster computers to “last setting” on “AC back”
• cluster8 – does not wake from standby (because of SP2, that was installed on the computer?)
• alpha8 – is not accessible by remote desktop
• Display date along with packages
• Maybe we should not put the computer running ArbiterServer into stand-by. It might be that this computer has nothing to do even though other computers are finishing last simulations or maybe the ArbiterServer’s JobServer was blacklisted.
• When the ArbiterServer is in stand-by it takes a long time for the Client to retrieve information.
• Why is the loading of server status so slow (user has to wait when he opens the client)
• Implement manual blacklisting
• Output messages with sense to the console for debugging
• Modify the FS to disable short filename generation: fsutil behavior set disable8dot3 1
1. if some old programs have problems on the cluster – revert this setting by: fsutil behavior set disable8dot3 0
• Run JobServer in the background on non-cluster computers:
1. Create a task that starts on idle and optionally dies if the computer becomes non-idle. In this task start the run_job_server_on_idle.bat.
2. Use the username/password of the currently logged in user to create the task because windows has issues:
1. Matlab won’t be killed if javaw is used
2. Does not work if the task is not created by the “console” user
3. Therefore: the background job will not work while nobody or somebody else is logged in
3. Careful with windows xp firewall, it might be blocking the ports 8188 and 8189, enable those.